feat(srt): check srt-slurm recipes against master images before sweep dispatch / feat(srt)：在派发 sweep 前检查 srt-slurm 配方与主配置镜像是否一致 - #3624

Description

Follow-up to #3567; the two PRs belong together. #3567 aligns the two srt-slurm recipes whose container had drifted from the master-config image. This PR adds the check that would have caught the drift: each sweep now binds every planned srt-slurm point to the recipe variants it would launch, and stops before dispatching any job if a point doesn't bind.

Problem. An srt-slurm point names its images in two places: the master config's image (plus PREFILL_IMAGE for TileRT) and the containers in the recipe it launches. An image bump has to edit both, as #3446 did, but nothing compares them before benchmark jobs start.

• Single-node points: infx/srt_slurm/single_node.py rejects a mismatch (Single-node SRT image: recipe/matrix ...) only after the benchmark job has started on its cluster runner, prepared an srt-slurm checkout and installed srtctl. [AMD] Update GLM-5.2 MI355X image to 20260924 daily / 更新 GLM-5.2 MI355X 镜像至 20260924 daily #3446's first sweep failed this way. So would every point of the two keys fix(srt): align B200 and MI355X recipe images with master configs / fix(srt)：对齐 B200 与 MI355X 配方镜像与主配置 #3567 fixes, because feat(srt): run AgentX on native srt-slurm #3428 ported their recipes from the legacy scripts as they were before the image bumps in config(dsv41flash): re-sweep MI355X on the first ROCm 10.0 nightly carrying vllm#58510 / 在首个包含 vllm#58510 的 ROCm 10.0 nightly 上重新扫描 MI355X #3420 and [Klaud Cold] Update dsv4-fp4-b200-sglang-agentic-hicache-mtp SGLang image to v0.5.20-cu130 / 将 dsv4-fp4-b200-sglang-agentic-hicache-mtp 的 SGLang 镜像更新至 v0.5.20-cu130 #3334.
• Multi-node points: nothing checks them. srtctl uses a recipe container that isn't in the job's containers: map as a literal image, so a stale recipe benchmarks a different image from the one the master config records.

#3567 first added a pytest scan for this. Review found that it couldn't gate run-sweep, compared image sets per recipe instead of the variant each point selects, skipped EVAL_CONFIG_FILE, and pinned checked-in config against the AGENTS.md test rules. #3567 dropped the scan, and this PR replaces it.

Changes

1. Validator. infx/srt_slurm/preflight.py (python -m infx.srt_slurm.preflight) reads the matrix on stdin. If every srt-slurm point binds, it writes the matrix unchanged to stdout; otherwise it prints each problem once with the points it affects and exits 1.
   • Single-node: it builds the environment that benchmark-tmpl.yml exports for the point and calls the runtime's own select_recipe, which must match exactly one variant on engine, model, image, precision, parallelism, GPU count, speculative decoding, concurrency and KV offloading. The check runs per point, so a recipe shared by keys with different images is checked against each key's image.
   • Multi-node: a point must set CONFIG_FILE, unless it is eval-only and sets EVAL_CONFIG_FILE. That is the recipe lanes.config_file launches; any other point would fail only on the GPU runner. The validator expands every CONFIG_FILE and EVAL_CONFIG_FILE a point references with srtctl's own override expansion. A reference that selects no variant is reported, because srtctl would submit nothing. In every variant:
     • model.container and each roles.<role>.container must be the point's image (in either registry spelling) or a container alias that configs/runners.yaml defines for the runner's cluster. A TileRT point must set PREFILL_IMAGE, and its prefill role must be exactly that image, the only prefill name the launcher stages.
     • A TileRT frontend.container_image or benchmark.container_image must be the decode image (or an alias) or PREFILL_IMAGE.
     • A declared identity.container.image must name the point's image.
   • Benchmark clients: in single-node and multi-node variants alike, a benchmark.container_image that names another tag of the point's image is reported, because srtctl would pull that literal instead of the staged image. An unrelated client image is allowed, and other frontends may pin an image of their own: the ATOM agentic recipes pin rocm/atom-dev@sha256:f00a… for their atomesh frontend.
   • Paths: a single-node srt-recipe must resolve inside benchmarks/single_node/srt-slurm-recipes, and a multi-node reference must be a recipes/ path that resolves inside benchmarks/multi_node/srt-slurm-recipes, so .. and absolute paths are rejected.
   • Runner inventory: it is read only when the matrix has multi-node points. If this tooling's schema cannot read it, the validator says so and exits 1.
2. Workflows.
   • run-sweep.yml: setup initializes the srt-slurm submodule and runs the validator after benchmark_schema --plan, before ci_priority. If the validator fails, setup fails, so canary-select and the benchmark jobs, which all require a successful setup, never start.
   • e2e-tests.yml (which trusted-external-sweep.yml and claude.yml also dispatch) and profile.yml: the validator and its srtctl come from the trusted tooling tree (PRIORITY_ROOT and its own srt-slurm submodule), and the measured tree is read only as data (--root "$MEASURED_ROOT"). The step always runs. require_launcher, earlier in the same step, already rejects revisions older than the Python launcher, and those are also the revisions whose runners.yaml this tooling cannot read.
3. Tests. infx/tests/srt_slurm/test_preflight.py has 36 cases with small temporary recipes and runner inventories, per the AGENTS.md test rules. They cover:
   • a one-sided image update through the CLI, and a runner inventory read only for multi-node points;
   • throughput and eval-only points with neither recipe, only EVAL_CONFIG_FILE, or only CONFIG_FILE;
   • a variant swap in a shared recipe, and an EVAL_CONFIG_FILE mismatch;
   • a recipe that selects no variant unless :base is named, and a variant without model.container;
   • configured and misspelled aliases, registry spellings, and runner labels, pools and bare cluster ids;
   • TileRT decode-only and prefill-only bumps, and a TileRT point without PREFILL_IMAGE;
   • benchmark clients on another tag, an unrelated image, or another spelling, and a pinned frontend;
   • absolute, .. and unprefixed recipe paths, and a missing recipe.

A sweep that selects the B200 key without #3567's fix stops at setup with:

srt-slurm recipe preflight found 1 problem(s) affecting 13 point(s):
  srt-recipe=benchmarks/single_node/srt-slurm-recipes/dsv4/sglang/b200-fp4-mtp/agentic.yaml: Expected exactly one matching single-node SRT recipe; override_dep8_hicache_c128: Single-node SRT image: recipe/matrix 'lmsysorg/sglang:v0.5.19-cu130' != 'lmsysorg/sglang:v0.5.20-cu130'; ...
    - dsv4_tp8_conc1_kvnone_spec-draft_model on cluster:b200-nscale
    ...
    - dsv4_tp8_conc160_kvdram-hicache_spec-draft_model on cluster:b200-nscale | eval-only

Changes since the first review

• e2e-tests.yml and profile.yml no longer import srtctl from the measured tree's submodule, which a PR could point at any code (Claude Code Review and the external review).
• The check no longer depends on whether the measured tree contains preflight.py.
• TileRT points are checked by role instead of skipped.
• A reference that selects no variant is reported instead of falling back to base.
• Recipe paths must stay inside their recipe trees.
• benchmark.container_image must follow the point's image unless it names an unrelated client image.
• A multi-node point without CONFIG_FILE is reported unless it is eval-only and sets EVAL_CONFIG_FILE (second review).
• The first commit carries a bilingual body; the branch is rebased onto main 1f60c15b5, and the second-review fix is its own bilingual commit.

Validation

All results below are on this branch rebased onto main 1f60c15b5.

• test_preflight.py: 36 passed. Each of 21 deliberate regressions in the validator fails at least one test, including skipping TileRT, treating the prefill role like the others, accepting zero variants, letting a throughput point launch EVAL_CONFIG_FILE, dropping either path check, and reading the runner inventory eagerly.
• ci.yml commands, run locally: Lint is clean, and Tests has 2,380 passed. Two test_slurm_cli cases fail only on the test host because it has a real sacct; they fail the same way on clean main.
• Workflow security: the workflow's exact zizmor command reports nothing on a clean checkout without submodules, as CI checks it out.
• Full generated matrix (1,755 points, 579 of them multi-node, about 11 s): 8 problems affecting 25 points, with no false positives.
  • dsv4-fp4-b200-sglang-agentic-hicache-mtp (12 points) is fixed by fix(srt): align B200 and MI355X recipe images with master configs / fix(srt)：对齐 B200 与 MI355X 配方镜像与主配置 #3567. dsv41flash-fp4-mi355x-vllm-agentic-dspark is now consistent on main after [AMD][MI355X] DSv4.1-Flash vLLM: enable ROCm paged MXFP4 sparse indexer / DSv4.1-Flash vLLM：启用 ROCm 分页 MXFP4 稀疏 indexer #3571.
  • glm5.2-fp8-mi325x-sglang-agentic-mtp (7 points) still drifts on main.
  • dsv4-fp4-mi355x-sglang-disagg-agentic-umbp-dspark (6 points, one per override): its recipe has run its benchmark client on lmsysorg/sglang-rocm:v0.5.19-rocm720-mi35x-20260907 since feat(srt): run AgentX on native srt-slurm #3428, while the master image is v0.5.20-rocm720-mi35x-20260923.
  • Every multi-node point sets the recipe its launcher selects. The TileRT key and the ATOM frontends pass.
• run-sweep setup replayed in a throwaway worktree with the workflow's exact commands under bash -e:
  • The B200 key without fix(srt): align B200 and MI355X recipe images with master configs / fix(srt)：对齐 B200 与 MI355X 配方镜像与主配置 #3567's fix exits 1 at the validator, before ci_priority, listing 14 points; with fix(srt): align B200 and MI355X recipe images with master configs / fix(srt)：对齐 B200 与 MI355X 配方镜像与主配置 #3567's recipe, the matrix passes through byte-identical.
  • Bumping only the glm5.3-fp8-mi355x-tilert-agentic master image from 0.1.6 to 0.1.7 exits 1 on model.container, roles.decode.container and frontend.container_image; bumping the recipe too passes.
  • minimaxm3-fp4-gb300-dynamo-vllm-agentic-mtp-disagg passes: its plan has 6 throughput points and 10 eval-only points. With its CONFIG_FILE lines removed and EVAL_CONFIG_FILE kept, the 6 throughput points fail and the eval-only points still pass.
• e2e-tests and profile: their preflight blocks, extracted verbatim from the workflows, ran under bash -e against a measured main checkout whose srtctl was replaced by a trap that exits on import, and which has no preflight.py. The trap never fired. The B200 drift was reported, and with fix(srt): align B200 and MI355X recipe images with master configs / fix(srt)：对齐 B200 与 MI355X 配方镜像与主配置 #3567's recipe the matrix passed unchanged. With the old PYTHONPATH the trap fired.

Notes for reviewers

• Please merge fix(srt): align B200 and MI355X recipe images with master configs / fix(srt)：对齐 B200 与 MI355X 配方镜像与主配置 #3567 first. Until it merges, a sweep that selects dsv4-fp4-b200-sglang-agentic-hicache-mtp fails at setup; it would fail at runtime anyway.
• glm5.2-fp8-mi325x-sglang-agentic-mtp and dsv4-fp4-mi355x-sglang-disagg-agentic-umbp-dspark will fail at setup when a sweep selects them, until their recipes are fixed. Fixing the dspark client image changes what the benchmark runs, so it needs its own changelog entry and sweep. Only the points a sweep plans are checked, so neither blocks unrelated PRs.
• single_node_environment mirrors how benchmark-tmpl.yml exports matrix fields to the job. A change to that mapping needs the same change here.
• run-sweep only triggers on PRs that edit perf-changelog.yaml, so this PR's own checks don't run the new setup step. The replays above stand in for that.
• There is no perf-changelog.yaml entry, because no benchmark config or recipe changes.
• All changed files are owned by @SemiAnalysisAI/core.

AI model disclosure

• Model/version: Claude Opus 5.5 (the identifier exposed by the Cursor agent runtime). No delegated agents were used.
• Role: designed and wrote the validator, workflow steps and tests to address the review findings on fix(srt): align B200 and MI355X recipe images with master configs / fix(srt)：对齐 B200 与 MI355X 配方镜像与主配置 #3567's earlier static test and on this PR, ran the local validation, and drafted this description. @chunfangamd reviewed the changes and chose the design.

Related Issue

No issue. Follow-up to #3567. Related: #3428, #3446.

Type of Change

• Bug fix
• New feature
• Configuration change
• Documentation update
• Other (please describe)

Checklist

• I have completed the AI model disclosure and kept it current
• I have tested my changes locally
• I have updated documentation if necessary
• For every change that can affect benchmark performance and every recipe addition or modification, I have appended a new entry to the physical end of inferencex-e2e/perf-changelog.yaml and have not edited historical entries
• Before merging via reuse, an authorized maintainer (OWNER/MEMBER/COLLABORATOR) has commented /use <run_id> (or the legacy /reuse-sweep-run) on this PR. Do this only once there is a final full sweep that is all green with evals passing, since after this comment the sweep label will no longer automatically kick off new sweeps. Remove and re-add the label to force one.